Papers with unsupervised sentence representation learning
Virtual Augmentation Supported Contrastive Learning of Sentence Representations (2022.findings-acl)
Copied to clipboard
| Challenge: | Despite profound successes, contrastive representation learning relies on carefully designed data augmentations using domain-specific knowledge. |
| Approach: | They propose a virtual augmentation supported Contrastive Learning of sentence representations . they approximate the neighborhood of an instance via its K-nearest in-batch neighbors . |
| Outcome: | The proposed model outperforms existing methods on a wide range of downstream tasks. |
Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning (P19-1)
Copied to clipboard
| Challenge: | Encoder-decoder models for unsupervised sentence representation learning discard decoder after training . decoded sentences are often used to make better predictions of words in a given sentence . |
| Approach: | They propose two types of decoding functions whose inverse can be easily derived without expensive inverse calculation. |
| Outcome: | The proposed models can learn good representations from encoders and decoders without expensive calculations. |
Bootstrapped Unsupervised Sentence Representation Learning (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to learn sentence representations rely on quality labeled data. |
| Approach: | They propose a Siamese Network which maximizes similarity between two augmented views of each sentence. |
| Outcome: | The proposed method outperforms state-of-the-art methods on STS and classification tasks. |
An Information Minimization Based Contrastive Learning Model for Unsupervised Sentence Embeddings Learning (2022.coling-1)
Copied to clipboard
| Challenge: | Recent contrastive learning methods keep positive pairs similar and push negative pairs apart, which leads to redundant information in sentence embeddings. |
| Approach: | They propose a contrastive learning approach which maximizes mutual information and minimizes the information entropy between positive and negative instances. |
| Outcome: | The proposed model outperforms all previous competitors on supervised and unsupervised tasks. |
OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding (2023.emnlp-main)
Copied to clipboard
Zhan Shi, Guoyin Wang, Ke Bai, Jiwei Li, Xiang Li, Qingjun Cui, Belinda Zeng, Trishul Chilimbi, Xiaodan Zhu
| Challenge: | Recent studies show that contrastive learning is effective in sentence representation learning . but, the surface structure bias is a problem in the current model . |
| Approach: | They propose to combine a sentence with a sub-semantic sentence to investigate the surface structure bias. |
| Outcome: | The proposed model achieves state-of-the-art on standard semantic textual similarity tasks using different pre-trained backbones. |
Clustering-Aware Negative Sampling for Unsupervised Sentence Representation (2023.findings-acl)
Copied to clipboard
| Challenge: | Using clustering-aware learning, in-batch negatives are often ignored in sentence representation learning. |
| Approach: | They propose a method that integrates cluster information into contrastive learning for unsupervised sentence representation learning. |
| Outcome: | The proposed method compares favorably with baselines on semantic textual similarity tasks. |
RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank (2023.acl-long)
Copied to clipboard
Jiduan Liu, Jiahao Liu, Qifan Wang, Jingang Wang, Wei Wu, Yunsen Xian, Dongyan Zhao, Kai Chen, Rui Yan
| Challenge: | Unsupervised sentence representation learning is one of the fundamental problems in natural language processing . contrastive learning methods fail to capture fine-grained ranking information among the sentences . |
| Approach: | They propose a novel approach for unsupervised sentence representation learning that integrates ranking consistency and ranking distillation with contrastive learning into a unified framework. |
| Outcome: | The proposed approach performs better over state-of-the-art models on STS and TR tasks. |
Improving Multi-Criteria Chinese Word Segmentation through Learning Sentence Representation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent Chinese word segmentation models tend to learn the segmentation knowledge through in-vocabulary words rather than understanding the meaning of the entire context. |
| Approach: | They propose a context-aware approach that incorporates unsupervised sentence representation learning over different dropout masks into the multi-criteria training framework. |
| Outcome: | The proposed approach achieves state-of-the-art (SoTA) performance on six of the nine CWS benchmark datasets and out-of vocabulary (OOV) recalls for eight of nine. |